Data storage space recovery

ABSTRACT

Storage space is reclaimed by cleaning and compacting data objects where data objects are stored by immutable storage. A storage area of which space needs to be reclaimed is identified. Active and stale data objects stored in a storage area are identified, and only active data objects are transferred to a shadow storage area from the storage area when recovering storage space. I/O operations can be fulfilled from the storage area and the shadow storage area. Compaction requests and I/O requests are throttled according to QOS parameters. Recovery of storage space does not cause a failure to meet performance requirements for any storage volume.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of priority under 35 U.S.C. §120 as a continuation of U.S. Non-Provisional patent application Ser.No. 15/811,001, entitled “DATA STORAGE SPACE RECOVERY,” filed on Nov.13, 2017, which is a continuation of U.S. Non-Provisional patentapplication Ser. No. 14/732,605, entitled “DATA STORAGE SPACE RECOVERY,”filed on Jun. 5, 2015, now U.S. Pat. No. 9,817,578, issued on Nov. 14,2017, which each of the above applications being hereby expresslyincorporated by reference in their entirety for all purposes.

BACKGROUND 1. Technical Field

The present invention generally relates to the field of data storageand, more particularly, to reclaiming data storage space.

2. Background Information

As the volume of data stored each year grows, a multitude of costsassociated with maintaining this data also grows. The cost of purchasingand powering storage devices is just a fraction of the total cost ofownership. To achieve the reliability, dataset sizes, and performancedemanded by modem big data applications, thousands of such devices mustbe interconnected and managed by complex data storage systems. The coststo purchase, install, and maintain such systems dominate the overallcost of storing any given unit of data. Reclaiming space used to storeobsolete, unreferenced data (aka, garbage collection) is an importanttechnique for controlling the growth of storage costs.

Traditional data storage systems eagerly delete from the underlyingbacking store in response to a user-level delete. Relaxing thisrequirement can lead to improved performance and simplified designacross a wide range of data storage systems, from individual hard disksand SSDs to storage arrays and distributed file systems. Yet datastorage systems that defer deletion can accumulate garbage data that isno longer referenced; such garbage consumes storage capacity anddecreases throughput. If left unchecked, the cost overhead andperformance degradation will become substantial.

Thus, better garbage collection techniques that can bound the totalamount of waste while incurring minimal maintenance overhead areimportant to data storage systems.

SUMMARY

In one example, data objects (DOs) are stored by immutable storage,meaning that data objects are not updated in place at storage nodes.Storage nodes can be divided into partitions and data objects can bestored sequentially within each partition. Data objects that havedifferent content are stored at different physical locations. Dataobjects are identified by their unique identifiers (DOIDs) which can becalculated based on their content. As a result, data objects that arerevised versions of previously stored data objects, are stored atdifferent locations and identified by different DOIDs. Entries for dataobjects are created and maintained in storage manager (SM) catalogs torecord actual storage locations, reference counts, and associationentries for data objects. A reference count indicates the number ofinstances that a data object is currently referenced. An associationentry indicates the number of instances that a data object is currentlyreferenced in each storage volume. Data objects which have referencecounts of zero are stale. Stale data objects can be expunged to recoverstorage space.

In one approach, storage space is reclaimed by cleaning and compactingdata objects. A storage area where space can be reclaimed is identified.Active and stale data objects in the storage area are identified, andonly active data objects are transferred from the old storage area to anew storage area (a shadow storage area). For active data objects thatare transferred to the shadow storage area, SM catalogs are updated withthe new storage locations in the shadow storage area. When the transferof active data objects is completed, the old storage area can be reusedfor additional storage. This recovery of storage space can be performedin conjunction with normal I/O operations without degrading I/Operformances. I/O operations can be fulfilled from the old storage areaand the new shadow storage area. Storage space recovery is initiated bycompaction (i.e., transfer) requests according to a policy that could bebased on available storage capacity threshold crossing or, alternately,based on time. Compaction requests and I/O requests are throttledaccording to QOS parameters. Accordingly, the recovery of storage spacedoes not cause a failure to meet performance requirements for anystorage volume.

Other aspects include components, devices, systems, improvements,methods, processes, applications, computer readable mediums, and othertechnologies related to any of the above.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level block diagram illustrating an environment forstoring data, according to one embodiment.

FIG. 2A is a high-level block diagram illustrating the access manager(AM) module from FIG. 1, according to one embodiment.

FIG. 2B is a high-level block diagram illustrating the storage manager(SM) module from FIG. 1, according to one embodiment.

FIG. 3 is a high-level block diagram illustrating an example of acomputer for use as one or more of the entities illustrated in FIG. 1,according to one embodiment.

FIGS. 4A-4F are diagrams illustrating immutable data object storage.

FIG. 5 is a diagram illustrating compaction of data objects using ashadow storage area.

FIGS. 6A-6F are diagrams illustrating compaction requests interspersedwith I/O requests.

DETAILED DESCRIPTION

The Figures (FIGS.) and the following description describe certainembodiments by way of illustration only. One skilled in the art willreadily recognize from the following description that alternativeembodiments of the structures and methods illustrated herein may beemployed without departing from the principles described herein.Reference will now be made to several embodiments, examples of which areillustrated in the accompanying figures. It is noted that whereverpracticable similar or like reference numbers may be used in the figuresand may indicate similar or like functionality.

FIG. 1 is a high-level block diagram illustrating an environment 100 forstoring data, according to one embodiment. An example of such anenvironment is described in further detail in U.S. patent applicationSer. No. 13/957,849, “High-Performance Distributed Data Storage Systemwith Implicit Content Routing and Data Deduplication” (case 24156),which is incorporated herein by reference in its entirety. Theenvironment 100 may be maintained by an enterprise, such as acorporation, university, or government agency. As shown, the environment100 includes a network 105, multiple application nodes 110, and multiplestorage nodes 130. While three application nodes 110 and three storagenodes 130 are shown in the embodiment depicted in FIG. 1, otherembodiments can have different numbers of application nodes 110 and/orstorage nodes 130. The environment 100 may also include a data locationtable (DLT) master system 170.

The network 105 represents the communication pathway between theapplication nodes 110 and the storage nodes 130. In one embodiment, thenetwork 105 uses standard communications technologies and/or protocolsand can include the Internet. Thus, the network 105 can include linksusing technologies such as Ethernet, 802.11, InfiniBand, PCI ExpressAdvanced Switching, etc. Similarly, the networking protocols used on thenetwork 105 can include multiprotocol label switching (MPLS),transmission control protocol/Internet protocol (TCP/IP), User DatagramProtocol (UDP), hypertext transport protocol (HTTP), simple mailtransfer protocol (SMTP), file transfer protocol (FTP), etc. The dataexchanged over the network 105 can be represented using technologiesand/or formats including image data in binary form (e.g. PortableNetwork Graphics (PNG)), hypertext markup language (HTML), extensiblemarkup language (XML), etc. In addition, all or some of the links can beencrypted using conventional encryption technologies such as securesockets layer (SSL), transport layer security (TLS), virtual privatenetworks (VPNs), Internet Protocol security (IPsec), etc. In anotherembodiment, the entities on the network 105 can use custom and/ordedicated data communications technologies instead of, or in additionto, the ones described above.

An application node 110 is a computer (or set of computers) thatprovides standard application functionality and data services thatsupport that functionality. The application node 110 includes anapplication module 113 and an access manager module (AM module) 115. Theapplication module 113 provides standard application functionality suchas serving web pages, archiving data, or data backup/disaster recovery.In order to provide this standard functionality, the application module113 issues write requests (i.e., requests to store data) and readrequests (i.e., requests to retrieve data). The access manager module115 handles these application write requests and application readrequests. The access manager module 115 is further described below withreference to FIG. 2A.

A storage node 130 is a computer (or set of computers) that stores data.The storage node 130 can include one or more types of storage devices137, such as hard disk, optical disk, flash memory, and cloud. Thestorage nodes 130 can be multi-layer and/or virtual, for example asdescribed in U.S. patent application Ser. No. 14/074,584, “Multi-layerData Storage Virtualization Using a Consistent Data Reference Model”(case 24159), which is incorporated herein by reference in its entirety.The storage node 130 includes a storage manager module (SM module) 135.The storage manager module 135 handles data requests received via thenetwork 105 from the access manager module 115 (e.g., access managerwrite requests and access manager read requests). This typicallyincludes reading and writing to storage devices 137, and bookkeepingfunctions such as updating tables of what data is located where. Thestorage manager module 135 is further described below with reference toFIG. 2B.

Typically, stored data is divided into storage volumes. Clients, throughthe application nodes 110, typically will request creation of storagevolumes for the storage of their data. Storage volumes may haveassociated performance requirements, such as minimum I/O operations persecond (IOPS), maximum IOPS and priority level. These may be specifiedin service level agreements, for example. The storage volumes areassigned to storage node(s) 130 and may be spread across multiplestorage nodes and storage devices within those storage nodes. Theperformance requirements may be different for different storage volumes,even though multiple storage volumes may be stored on the same storagenode or on the same storage device.

Performance requirements are often expressed as quality of service (QOS)parameters. Examples of QOS parameters include service level agreementsor performance specified in service level agreements, I/O priority, andminimum and maximum input/output operations per second (IOPS). Servicelevel agreements may specify minimum down time, maximum recovery time,maximum latency, etc. I/O priority is typically some sort of priorityorder. For example, storage volumes with I/O priority 1 may havepriority over those with I/O priority 2, which have priority over thosewith I/O priority 3, etc. Minimum IOPS typically is the minimumthroughput which must be available from the data storage environment atall times. In other words, the data storage environment must be able tosupport a situation where the storage volume is continuously operatingat the minimum IOPS rate. Maximum IOPS typically specifies a burst ratewhich must be available from the data storage environment. If thestorage volume has burst activity up to the maximum IOPS rate, the datastorage environment must support that burst activity. However, it is notrequired to support burst activity above this rate nor is it required tosupport this rate for long periods of time or for continuous operation.

The data itself is represented as data objects (DO). Unlike many typesof storage, data objects can vary widely in size and characteristics.Examples of data objects include emails, images, audio files, videofiles, system logs, key-value store data, volume blocks, unstructureddata, data structured specifically for certain database or otherapplications, etc. An application data identifier (ADID) is theidentifier used by the application module 113 to refer to the dataobject within the application. The application data identifier can be,for example, a file name, an object name, or a range of blocks. A dataobject identification (DOID) is a unique address that is used as theprimary reference for placement and retrieval of the data object withinthe storage nodes 130. The access manager module 115 translates betweenADID and DOID. The DLT master system 170 maintains tables indicatingwhich partitions of data objects are stored at which storage nodes,including indicating the locations of redundant storage if so desired.Relevant tables from the DLT master system 170 may be mirrored on theaccess manager modules 115 and/or storage manager modules 135.

Basic read/write operation of system 100 is as follows. With respect towriting data to storage, the application module 113 sends an applicationwrite request 151 to the access manager module 115. The applicationwrite request includes a data object and an application data identifier.The access manager module 115 determines 152 a pending DOIDcorresponding to the application data identifier and determines 152 oneor more storage nodes 130 on which the DO should be stored. The AMmodule 115 sends an access manager (AM) write request 153 to the storagenodes 130. The AM write request includes the data object and the DOID.In step 154, the SM module 135 finalizes the pending DOID, stores the DOand updates a local catalog by adding an entry mapping the DO'sfinalized DOID to the actual storage location on the storage device 137where the DO was stored. The SM module 135 sends an acknowledgment 155to the AM module 115, which does some local bookkeeping 156 and sends anacknowledgment 157 to the application module 113.

Reading data from storage operates similarly. The application module 113sends an application read request 161 to the AM module 115. Theapplication read request includes an application data identifier. Theaccess manager module 115 determines 162 the DOID corresponding to theapplication data identifier and determines 162 one or more storage nodes130 on which the DO is stored. The AM module 115 sends an access manager(AM) read request 163 to the appropriate storage nodes 130. The AM readrequest includes the DOID. In step 164, the SM module 135 determines theactual storage location from the DOID and retrieves the data object fromthe corresponding storage device 137. The SM module 135 sends 165 thedata object to the AM module 115, which does some local bookkeeping 166and sends 167 the data object to the application module 113.

Note that in both read and write operations, while the applicationmodule 113 refers to data using application data identifiers, DOIDs areused by the AM module 115 and the SM module 135. The AM module 115converts between application data identifiers and DOIDs. In addition,the DOIDs are unique for each data object and preferably are calculatedbased on the content of the data object.

Also note that many application modules 113 may be requesting read andwrite of many data objects from many storage volumes spanning manystorage nodes 130 and storage devices 137. These I/O requests preferablyare prioritized in a manner that meets performance requirements for thedifferent storage volumes. Examples of different approaches aredescribed in U.S. patent application Ser. No. 14/229,690, “EfficientScalable I/O Scheduling” (case 25875) and Ser. No. 14/613,779, “DynamicStorage Tiering Based on Performance SLAs” (case 27845), which areincorporated herein by reference in their entirety. In addition,redundancy or replication may also be implemented in the data storage,for example as described in U.S. patent application Ser. No. 14/495,431,“Assigning Storage Responsibility in a Distributed Data Storage Systemwith Replication” (case 26555), which is incorporated herein byreference in its entirety.

FIG. 2A is a high-level block diagram illustrating the access managermodule 115 from FIG. 1, according to one embodiment. The AM module 115includes a number of functional modules and a repository 220 whichcontains data. In this example, the functional modules include a DOIDgeneration module 211, an access manager (AM) storage location module212, an access manager (AM) storage module 213, and an access manager(AM) retrieval module 214.

The repository 220 stores a virtual volume catalog 221 and a datalocation table 222 (sometimes referred to as the AM data location table222). The virtual volume catalog 221 stores mappings between applicationdata identifiers (ADIDs) and data object identifications (DOIDs). Oneapplication data identifier is mapped to one DOID. The DOID is a uniqueaddress that is used as the primary reference for placement andretrieval of a data object. Updates to the virtual volume catalog 221may be stored by one or more storage nodes 130. This embodiment providesa redundant, non-volatile, consistent replica of the virtual volumecatalog 221 data within the environment 100. In this embodiment, when aaccess manager module 115 is initialized or restarted, the appropriatecopy of the virtual volume catalog 221 is loaded from a storage node 130into the access manager module 115. In one embodiment, the storage nodes130 are assigned by volume ID (i.e., by each unique storage volume), asopposed to by DOID. In this way, all updates to the virtual volumecatalog 221 will be consistent for any given storage volume.

The data location table 222 stores data object placement information,such as mappings between DOID Locators (“DOID-Ls” or “tokens”, the first4 bytes of DO IDs in this example) and storage nodes. One DOID-L ismapped to one or more storage nodes (indicated by storage nodeidentifiers). The AM data location table 222 is the relevant portion ofthe global mapping information maintained by the DLT master system 170.That is, the DLT master system 170 maintains data location tables forthe entire system, while the AM data location table 222 is theinformation relevant to the specific access manager module 115. The oneto many mapping can be used to implement redundancy. A storage nodeidentifier is, for example, an IP address or another identifier that canbe directly associated with an IP address. In one embodiment, themappings are stored in a relational database to enable rapid access. Forexample, see U.S. patent application Ser. No. 14/495,431, “AssigningStorage Responsibility in a Distributed Data Storage System withReplication” (case 26555), which is incorporated herein by reference inits entirety.

In one embodiment, a DOID-L is a four-byte value that can range from [0000 00 00] to [FF FF FF FF], which provides more than 429 millionindividual data object locations. Since the environment 100 willgenerally include fewer than 1000 storage nodes, a storage node would beallocated many (e.g., thousands of) DOID-Ls to provide a good degree ofgranularity. In general, more DOID-Ls are allocated to a storage node130 that has a larger capacity, and fewer DOID-Ls are allocated to astorage node 130 that has a smaller capacity.

The DOID generation module 211 takes as input a data object, generates adata object identification (DOID) for that object, and outputs thegenerated DOID. In one embodiment, the DOID generation module 211generates the DOID in part based on hashing the content of the dataobject. The AM storage location module 212 takes as input a DOID,determines the one or more storage nodes associated with the DOID, andoutputs the one or more storage nodes (indicated by storage nodeidentifiers). The AM storage module 213 takes as input an applicationwrite request, processes the application write request, and outputs aaccess manager (AM) write acknowledgment. The AM retrieval module 214takes as input an application read request, processes the applicationread request, and outputs a data object.

FIG. 2B is a high-level block diagram illustrating the storage managermodule 135 from FIG. 1, according to one embodiment. The SM module 135includes a number of functional modules and a repository 240 whichcontains data. In this example, the functional modules include storagelocation module 231, a storage manager (SM) storage module 232, astorage manager (SM) retrieval module 233, an orchestration managermodule 234, and a storage manager (SM) recovery module 235.

The repository 240 stores a SM catalog 241. The SM catalog 241 storesmappings between data object identifications (DOIDs) and actual storagelocations (e.g., on hard disk, optical disk, flash memory, and cloud).One DOID is mapped to one actual storage location within a storage node.For a particular DOID, the data object associated with the DOID isstored at the actual storage location. The same data object may bestored on multiple different storage nodes to implement redundancy. Insome implementations, the repository 240 may also store a storagemanager data location table (not shown in FIG. 2B). Similar to the AMdata location table, the SM data location table may contain the datafrom the DL T master system 170 which is relevant to this particularstorage node 130.

The SM storage location module 231 takes as input a DOID, determines theactual storage location associated with the DOID, and outputs the actualstorage location. The SM storage module 232 takes as input an AM writerequest, processes the AM write request, and outputs a SM writeacknowledgment. This processing includes resolving hash collisions. TheSM retrieval module 233 takes as input an AM read request, processes theAM read request, and outputs a data object.

The orchestration manager module 234 performs storage allocation andtuning among the various storage nodes 130. Only one storage node 130within the environment 100 needs to include the orchestration managermodule 234. However, in one embodiment, multiple storage nodes 130within the environment 100 (e.g., four storage nodes) include theorchestration manager module 234. In that embodiment, the orchestrationmanager module 234 runs as a redundant process.

The SM compaction module 235 cleans and compacts stale data objects toreclaim storage space on the storage device 137. In one approach, the SMcompaction module 235 operates periodically to compact data objectsstored over a range of addresses. In one embodiment, a storage node isdivided into partitions and storage space is recovered on aper-partition basis. For each partition, the SM compaction module 235may iterate over the SM catalog 241 to determine a ratio of stale toactive data objects and prioritize the partitions based on their ratios.Partitions having higher ratios of stale data objects are prioritizedover those having lower ratios.

The SM compaction module 235 manages compaction requests through the useof compaction queues. In one approach, the SM compaction module 235processes the data objects one at a time: consider a data object,determine whether the data object is active, transfer the data object toa shadow storage area if it is active, then consider the next dataobject. Alternatively, the SM compaction module 235 determines all dataobjects in a partition at once, making a list of which data objects areactive and then transferring them one by one.

The SM compaction module 235 reclaims storage space on the storagedevice 137 while SM storage module 232 and SM retrieval module 233 areprocessing I/O requests. I/O requests and compaction requests areintermingled such that I/O performance is not degraded. Processing ofI/O requests may be temporarily paused when a compaction operation withrespect to an active data object is being processed. Compaction requestsand I/O requests preferably are throttled according to QOS parameters.Accordingly, SM compaction module 235 does not cause a failure to meetperformance requirements for any storage volume. The SM compactionmodule 235 may initiate storage space recovery according to a policythat is based on available storage capacity or one that is based ontime. For example, when the available storage space for a storage device137 (or for a partition) falls below a threshold (e.g., 20-30%) of thetotal storage capacity, the SM compaction module 235 initiates storagespace recovery.

Storage nodes 130 can be added to (and removed from) the environment 100dynamically. Adding (or removing) a storage node 130 will increase (ordecrease) linearly both the capacity and the performance of the overallenvironment 100. When a storage node 130 is added, data objects areredistributed from the previously-existing storage nodes 130 such thatthe overall load is spread evenly across all of the storage nodes 130,where “spread evenly” means that the overall percentage of storageconsumption will be roughly the same in each of the storage nodes 130.In general, the orchestration manager module 234 balances base capacityby moving DOID-L segments from the most-used (in percentage terms)storage nodes 130 to the least-used storage nodes 130 until theenvironment 100 becomes balanced.

Recall that the data location tables (whether at the DLT master system170, in the access manager module 115 or in the storage manager module135) stores mappings (i.e., associations) between DOID-Ls and storagenodes. The aforementioned data object redistribution is indicated in thedata location tables by modifying specific DOID-L associations from onestorage node 130 to another. Once a new storage node 130 has beenconfigured and the relevant data object has been copied, an accessmanager module 115 will receive a new data location table reflecting thenew allocation. Data objects are grouped by individual DOID-Ls such thatan update to the data location table in each access manager module 115can change the storage node(s) associated with the DOID-Ls. Note thatthe existing storage nodes 130 will continue to operate properly usingthe older version of the data location table until the update process iscomplete. This proper operation enables the overall data location tableupdate process to happen over time while the environment 100 remainsfully operational.

In one embodiment, the orchestration manager module 234 also controlsthe registration of new storage volumes, for example as described inU.S. patent application Ser. No. 14/673,723, “Volume Admission Controlfor High-Performance Distributed Data Storage” (case 28716), which isincorporated herein by reference in its entirety.

The DLT master system 170 maintains a master data location table (mDLT)and updates the mDLT as necessary. For example, the DL T master system170 updates the mDLT when one or more storage nodes 130 come online,when one or more storage nodes 130 go offline, when weights of one ormore storage nodes 130 change, or any combination thereof. The DLTmaster system 170 sends the updated mDLT 530 to the application nodes110 and the storage nodes 130.

As described previously, DOID Locators (aka, tokens) correspond topartitions of data objects. The mDLT maintains the assignment of tokensto storage nodes. A token may be assigned to more than one storage node,for example to implement redundancy. In that case, different storagenodes may have different levels of responsibility for a particulartoken. For example, the mDLT may indicate that a particular storage nodehas primary responsibility for a taken, while other storage nodes havesecondary (i.e., redundancy) responsibility for the same token. The DL Tmaster system 170 preferably deterministically allocates secondary andprimary responsibilities based on the storage nodes' capabilities. Forexample, see U.S. patent application Ser. No. 14/495,431, “AssigningStorage Responsibility in a Distributed Data Storage System withReplication” (case 26555), which is incorporated herein by reference inits entirety.

FIG. 3 is a high-level block diagram illustrating an example of acomputer 300 for use as one or more of the entities illustrated in FIG.1, according to one embodiment. Illustrated are at least one processor302 coupled to a chipset 304. The chipset 304 includes a memorycontroller hub 320 and an input/output (I/O) controller hub 322. Amemory 306 and a graphics adapter 312 are coupled to the memorycontroller hub 320, and a display device 318 is coupled to the graphicsadapter 312. A storage device 308, keyboard 310, pointing device 314,and network adapter 316 are coupled to the I/O controller hub 322. Otherembodiments of the computer 300 have different architectures. Forexample, the memory 306 is directly coupled to the processor 302 in someembodiments.

The storage device 308 includes one or more non-transitorycomputer-readable storage media such as a hard drive, compact diskread-only memory (CD-ROM), DVD, or a solid-state memory device. Thememory 306 holds instructions and data used by the processor 302. Thepointing device 314 is used in combination with the keyboard 310 toinput data into the computer system 300. The graphics adapter 312displays images and other information on the display device 318. In someembodiments, the display device 318 includes a touch screen capabilityfor receiving user input and selections. The network adapter 316 couplesthe computer system 300 to the network 105. Some embodiments of thecomputer 300 have different and/or other components than those shown inFIG. 3. For example, the application node 110 and/or the storage node130 can be formed of multiple blade servers and lack a display device,keyboard, and other components.

The computer 300 is adapted to execute computer program modules forproviding functionality described herein. As used herein, the term“module” refers to computer program instructions and/or other logic usedto provide the specified functionality. Thus, a module can beimplemented in hardware, firmware, and/or software. In one embodiment,program modules formed of executable computer program instructions arestored on the storage device 308, loaded into the memory 306, andexecuted by the processor 302.

FIGS. 4A-4F are diagrams illustrating immutable data object storagewhere data objects are not updated in place at storage nodes. That is,different versions of a data object correspond to different DOIDs andare stored at different physical locations at a storage node. If anexisting data object stored at a physical location is changed, it is notreplaced by the new version. Rather, the new version will be stored at adifferent physical location.

In various embodiments, read and write requests to storage nodes 130 aremanaged by I/O queues. As illustrated in FIG. 4A, an I/O queue 402includes I/O operations (e.g., read, write, delete) requested byapplication modules 113. FIG. 4A shows an I/O queue 402 with 11 I/Orequests, starting with “Vol 1: Write data object A” and ending with“Vol 1: Write (data object) A′.” Data objects are organized into storagevolumes and I/O operations relate to different data objects anddifferent storage volumes. When a write request is received to store adata object, the data object is stored at the storage node 130 accordingto the write request. For example, the I/O operation “Vol 1: Write dataobject A” is a request by storage volume 1 to store a data object A.“Vol 1: Write B” is a request by storage volume 1 to store a differentdata object B. “Vol 2: Write A” is a request by storage volume 2 tostore data object A. “Vol 1: Delete B” is a request by storage volume 1to delete data object B. In these examples, data object B is differentthan data object A (i.e., data objects A and B have differentapplication data identifiers), but data object A′ is a modified versionof data object A (i.e., data objects A and A′ have the same applicationdata identifier but different contents and therefore different DOID).FIGS. 4A-4F illustrate processing of the requests in I/O queue 402.

FIG. 4A illustrates processing of the first three requests, as indicatedby the horizontal line. The SM catalog 241 stores mappings between DOIDsand actual storage locations. When a new data object is stored, the SMcatalog 241 is updated with the mapping between the DOID and the actualstorage location of the data object. For example, consider the first I/Orequest 410, which is a request by storage volume 1 to store a dataobject A. The data object A is stored at addr01 of the storage device137, as shown by arrow 411. When the data object A is stored, the SMcatalog 241 is updated with the mapping 412 between the DOID 413 of thedata object A, which is denoted as DOID[A], and the actual storagelocation 414 addr01 of the data object A. In addition to storingmappings between DOIDs and actual storage locations of data objects, theSM catalog 241 also stores reference counts 415 and association entries416 for data objects. A reference count is a count of a number ofinstances of a data object that is currently referenced. The associationentry is per storage volume and indicates a number of instances of adata object that is currently referenced in each storage volume. In thisexample, because the data object A is new and stored for the first time,the reference count 415 of the data object A is 1, indicating that dataobject A is currently being referenced once across all storage volumes.The association entry 416 for data object A is “vl,lx”, which indicatesthat storage volume 1 references data object A 1 time.

The next I/O request 419 is processed similarly. It is a request bystorage volume 1 to store data object B. Data object Bis new, so it isstored 420 at a new storage location addr02, and a corresponding entry421 is added to the SM catalog 241. I/O request 422 is processedsimilarly. Data object C is stored 423 at a new storage location addr03,and a corresponding entry 424 is added to the SM catalog 241. In oneapproach, data objects are stored sequentially and entries are addedsequentially in SM catalogs. Because the DOIDs are unique and determinedbased on content of data objects, the DOIDs of the data objects A,B,Care different. Within a storage node 130, because one DOID is mapped toone actual storage location, the actual storage location of the dataobject can be determined from the SM catalog 241 based on the DOID.

FIG. 4B illustrates the processing of I/O operation 430, which is awrite request by storage volume 2 to store data object A. However, thedata object A has already been stored by the I/O operation 410, which isa write request by storage volume 1 to store the data object A. Forexample, perhaps a file is attached to an email application and openedby different users using different email applications but using the samebackend storage. Therefore, there is no need to store data object A inthe storage device 137, because it is already stored 411. There is alsono need to add an entry for data object A in the SM catalog 241, becauseit already has an entry 412. However, the reference count 431 and theassociation entry 432 are updated. The reference count 431 isincremented to 2 because the data object A is currently being referencedtwice: once by storage volume 1 and once by storage volume 2. Theassociation entry 432 is updated to indicate that the data object A iscurrently being referenced once in storage volume 1 and once in storagevolume 2.

FIG. 4C illustrates the processing of the next four I/O operations,beginning with I/O operation 440. I/O request 440 is processed the sameas described in FIG. 4A, resulting in data object D stored at addr04 anda corresponding entry added to the SM catalog 241.

I/O request 441 is another write request by storage volume 1 to storedata object A. Because the data object A is stored at the storage device137 from the previous write requests, the data object A is not re-storedat the storage device 137. However, in the SM catalog 241, the referencecount 442 and the association entry 443 associated with DOID 424 areupdated. The reference count 442 is incremented to indicate that thetotal number of instances the data object A currently being referencedis 3: twice in storage volume 1 and once in storage volume 2. Theassociation entry 443 is updated to indicate that data object A is nowreferenced 2× by storage volume 1.

I/O requests 444 and 445 are processed the same as before, resulting innew data objects E and F stored at the storage device 137 andcorresponding entries in the SM catalog 241.

FIG. 4D illustrates processing a request 450 to delete data object B.The data object B is not physically deleted from the storage device 137.Notice that there is no change in the storage device 137 relative toFIG. 4C. However, the SM catalog 241 is updated. In entry 421, thereference count 451 and the association entry 452 for the data object Bare updated. The reference count 451 for the data object B isdecremented to 0, which indicates that the data object B is notcurrently being referenced by any storage volume. The association entry452 is also updated to indicate that the number of instances of dataobject B being currently referenced in storage volume 1 is 0. Dataobjects that are not currently being referenced are referred to as staledata objects. The SM catalog 241 may further record a timestamp when adata object becomes stale and/or the time period for which the dataobject has been stale.

FIG. 4E illustrates an I/O operation 460, which is a write request bystorage volume 1 to store the data object C′. The data object C′ is arevised version of a previously stored data object C. The systemdetermines that data object C′ is a revised version of data object Cbecause they use the same application data identifier. Otherwise, itwould treat the incoming data object as a new data object G. However,the content of the data objects C′ and C are different and, thus, theDOID of data object C′ is different from the DOID of data object C.Accordingly, the data object C′ is stored 461 at a new address addr07,rather than overwriting the data object C at addr03. A new entry 462 isalso added to the SM catalog 241, mapping DOID[C′] to the actual storagelocation addr07 of the data object C′.

However, because data object C′ is a revised version of data object C,the entry 424 for data object C is also revised. The reference count 463is decremented and the association entry 464 is updated to indicate thatthe data object C is not currently being referenced in storage volume 1.Because data object C′ is a revised version of data object C, therequest to write C′ is effectively interpreted as a request to write C′coupled with a request to delete C.

FIG. 4F illustrates an I/O operation 480 which is a write request bystorage volume 1 to store the data object A′. The data object A′ is arevised version of the data object A. This request is effectivelyinterpreted as a request to write A′ coupled with a request to delete A.This situation is similar to FIG. 4E, except that data object A isreferenced multiple times. The writing of A′ is straightforward andresults in storing 481 data object A′ with a corresponding entry 482 inthe SM catalog 241.

The entry 412 for data object A in SM catalog 241 is also updated. Priorto this I/O operation, the reference count 483 and association dataentry 484 of the data object A indicated that the data object A wasbeing referenced 3× in total: twice in storage volume 1 and once instorage volume 2. As a result of this I/O operation 480, which is arequest by storage volume 1, the reference count 483 is decremented andthe association data entry 484 is updated to indicate that the dataobject A is currently being referenced 2× in total: once in storagevolume 1 and once in storage volume 2.

FIGS. 4A-4F show examples of requests to write and to delete dataobjects. Normal I/O operation typically will also include requested toread data objects. These are not shown in FIGS. 4A-4F for purposes ofclarity. For example, in FIG. 4F, if there was a request to “Read D,”the actual storage location of data object D (addr04) would bedetermined from the SM catalog 241 and then data object D would beretrieved from that storage location.

FIG. 4F shows the SM catalog 241 and storage device 137 after processingof all I/O operations in the I/O queue 402. Data objects for whichreference counts are zero are stale. Data objects for which referencecounts are non-zero are active. In FIG. 4F, data object B and C arestale. The spaced used to store stale data objects can be recovered bycompaction processes that keep only active data objects. In someembodiments, compaction processes remove any object data object that isstale. In other embodiments, data objects must be stale for a certaingrace period of time before they are removed by compaction processes.The grace period may be determined from a lifecycle policy associatedwith a storage node or a storage volume. In various embodiments,compaction processes use shadow storage areas and throttle betweencompaction and I/O operations (or requests) to reclaim physical storagespace without degrading I/O performance and increasing average latencyfor I/O operations. Compaction processes are described in detail withreference to FIGS. 5 through 6D.

FIG. 5 illustrates a compaction process using a shadow storage area. InFIG. 5, the SM catalog 241 (before) shows the state of the SM catalog241 at the end of FIG. 4F, which is before the compaction process isrun. The SM catalog 241 (after) shows the SM catalog 241 after thecompaction process is completed. The storage area (frozen) 504 is wherethe data objects were stored at the end of FIG. 4F, which is before thecompaction process is run. During the compaction process, the activedata objects are transferred from this frozen storage area 504 to ashadow storage area 506. Stale data objects are not transferred. Afterthe transfer is complete, the frozen storage area 504 may be releasedfor re-use. Note that SM catalog 241 (before) and SM catalog 241 (after)show the SM catalog 241 before and after the compaction process.However, the frozen storage area 504 and the shadow storage area 506 arenot before and after versions of the same storage area. Rather, they aretwo different storage areas within the storage device 137.

The compaction process is implemented using a compaction queue 502,which includes compaction requests for the data objects being compacted.Each data object included in the compaction request is processed byreferencing the SM catalog 241 to determine whether the data object isactive or stale. Active data objects are transferred from the storagearea 504 to the shadow storage area 506 whereas stale data objects arenot. For active data objects that have been transferred to the shadowstorage area, the SM catalog is updated with their new actual storagelocations in the shadow storage area. SM catalogs are updated to removestale data objects.

For example, the first request 510 in compaction queue 502 is “CompactA.” The data object A is determined to be active because its referencecount in SM catalog entry 511 indicates it is currently referenced.Therefore, the data object A is copied from the frozen storage area 504to the shadow storage area 506, where it is stored at the actual storagelocation addr20. The SM catalog 241 is then updated with the actualstorage location 512, which in this case is addr20.

The next request 513 is “Compact B.” However, data object B is stalebecause its reference count is 0. Therefore, data object B is not copiedto the shadow storage area 506. The same is true for the compactionrequest 514 “Compact C.”

Data object D is active, so the compaction request 515 “Compact D”results in copying of data object D to the shadow storage area 506 ataddr21. The SM catalog 241 is also updated to reflect the new actualstorage location. This process is repeated for the remaining compactionrequests in the compaction queue 502. FIG. 5 shows the SM catalog 241and storage device 137 after all compaction requests in the compactionqueue 502 have been processed. The shadow storage area 506 stores onlythe active data objects, and the SM catalog 241 is updated to reflectthis.

During the compaction process of a storage area, the original storagearea 504 is frozen such that no data objects can be written into thestorage area but data objects which have not yet been transferred can beread from the storage area. Once the compaction queue 502 is completelyprocessed, the original storage area 504 is freed up for re-use andoperation cuts over fully to the shadow storage area 506.

FIG. 5 illustrates processing of a compaction queue 502. In productionoperation, compaction requests to recover physical storage space (asillustrated in FIG. 5) are typically processed interspersed with I/Orequests to read, write and delete data objects (as illustrated in FIG.4). The processing of compaction requests and I/O requests preferablyare throttled to have acceptable performance degradation to I/Ooperations. For a storage node, compaction requests preferably do notconsume more than 10% of the maximum I/O operations per second. For astorage node, compaction requests preferably do not increase averagelatency for I/O operations by more than 10%. In various embodiments,data objects are organized into storage volumes, and the compactionprocess preferably does not cause a failure to meet performancerequirements for any storage volume.

The rate of performing space recovery can be adjusted. For a storagenode, the ratio of I/O operations to compaction operations can beadjusted. In one approach, each storage node is divided into partitions(or tokens) and storage space is recovered on a per-token basis. Eachstorage manager has a set of tokens assigned to it, and the data objectsbelonging to that token will be stored in that SM. Further, each tokenhas a unique mapping to storage devices, meaning objects belonging tothat token are stored in that storage device. Each token also has acatalog and an active storage area/partition. The rate of compactionoccurring at the storage node can be throttled by adjusting the numberof tokens that are undergoing compaction at any given time. Thethrottling can be done automatically or manually, for example by anadministrator for the storage environment.

FIGS. 6A-6F are diagrams illustrating compaction requests interspersedwith I/O requests. Because compaction requests are processedinterspersed with I/O operations, I/O operations may request to read andwrite data objects that are among the data objects scheduled forcompaction. Active data objects that are to be transferred from thefrozen storage area to the shadow storage area, but which have not yetbeen transferred, are read from the frozen storage area. Active dataobjects that have already been transferred (copied) to the shadowstorage area are read from the shadow storage area.

FIG. 6A shows the state of a storage system before compaction begins.The SM catalog 241 and storage device 137 are the same as at the end ofFIG. 4F (or the beginning of FIG. 5). In this example, there is acompaction queue 502 of compaction requests, which is the same as inFIG. 5. However, there is also an I/O queue 402 of I/O requests whichoccur during the compaction process. The compaction requests and I/Orequests are combined into a system queue 602 and processed from thesystem queue 602, as illustrated in FIGS. 6B-6E.

In FIG. 6B, the first five compaction requests and the first two I/Orequests have been combined into the system queue 602. FIG. 6Billustrates processing of these requests. The system begins byprocessing the compaction requests 610, as described in FIG. 5. Activedata objects A, D and E are copied from the frozen storage area 504 tothe shadow storage area 506. The SM catalog 241 is updated to reflectthe new storage locations for data objects A, D and E. Stale dataobjects B and C are not copied. Because compaction requests 680 have notbeen processed, active data objects F, C′ and A′ have not yet beencopied from the frozen storage area 504 to the shadow storage area 506.The SM catalog 241 still indicates actual storage locations of dataobjects F, C′, and A′ in the frozen storage area 504. For clarity,addresses in the shadow storage area will be indicated by italics andindentation in the SM catalog 241.

The I/O request 611 is a read request from storage volume 1 to read thedata object A. Because the data object A has been copied to the shadowstorage area, the entry 612 in the SM catalog 241 indicates that itsactual storage location is addr20. Data object A is thus read from theshadow storage area 506. The I/O request 613 is a read request fromstorage volume 1 to read the data object C′. Because the compactionrequest for data object C′ has not yet been processed, the entry 614 inSM catalog 241 indicates that its actual storage location is addr07. Thedata object C′ is read from the actual storage location addr07, which isin the frozen storage area 504.

FIG. 6C shows processing of the next one compaction request and the nextthree I/O requests. I/O request 620 is a write request from storagevolume 1 to store a new data object G. When processing a compactionrequest to recover a storage area, I/O operations to write new dataobjects are fulfilled using the shadow storage area rather than thefrozen storage area. The new data object G is stored in the shadowstorage area 506 at the actual storage location addr23. The SM catalog241 is updated with a new entry 621 for the data object G. The SMcatalog 241 is further updated to include the reference count and theassociation entry for the data object G.

The next I/O operation 623 is a write request to store data object B.Writes are always made to the shadow storage area. In this case, dataobject B is stored in the shadow storage area 506 at addr24. The SMcatalog entry 624 for the data object Bis updated with the new actualstorage location, reference count and association entry. It does nothappen in this example, but if a “Compact B” request was encounteredafter I/O operation 623, data object B would not have to be transferredbecause it is already in the shadow storage area.

During compaction, I/O operations to delete data objects follow the sameprocess as described with respect to FIG. 4. For example, the I/Ooperation 626 requests to delete the data object C′. As with a normaldeletion process, the entry 627 for data object C′ is updated.Specifically, the reference count 628 for data object C′ is decrementedand the association entry 629 for data object C′ is correspondinglyupdated. The data object C′ is an active data object that has not yetbeen transferred from the frozen storage area 504 to the shadow storagearea 506. However, this does not affect the deletion process.

Finally, compaction request 631 is processed. Data object F isdetermined to be active, so it is copied to the shadow storage area 506at addr25. The corresponding entry 632 in the SM catalog 241 is updated.

FIG. 6D shows processing of the next one compaction request and the nexttwo I/O requests. Write request 640 stores data object H at addr26 andcreates a corresponding entry 641 in the SM catalog 241.

The compaction request 643 could be handled in several ways. In oneapproach, it is determined that data object C′ is now stale. As such, itis not transferred to the shadow storage area 506 and the SM catalogentry 644 remains unchanged. This is the approach shown in FIG. 6D.However, in an alternate approach, the data object C′ is copied from thestorage area 504 to the shadow storage area 506 even though the currentreference count is 0. This might occur, for example, if the stalenessdetermination is made before the I/O request to “Delete C′” isprocessed. In this approach, the stale data object C′ would be copied tothe shadow storage area 506 and the entry 644 would be updatedaccordingly.

I/O request 646 writes a new data object A″. Data object A″ is amodified version of data object A′. Therefore, this request isimplemented as a request to delete data object A′ and a request to writedata object A″. The data object A″ is written into the shadow storagearea 506 and the SM catalog 241 is updated with the mapping 647 betweenDOID[A″ ] and the actual storage location addr27. Deleting data objectA′ is implemented by decrementing the reference count 648 and updatingthe association entries.

FIG. 6E shows processing of the final compaction request 650, “CompactA′.” Data object A′ has a reference count of 0, so this situation issimilar to the request “Compact C′” in FIG. 6D. That is, the data objectA′ is not copied to the shadow storage area 506.

FIG. 6F shows clean up at the end of the compaction process. Space inthe SM catalog 241 is recovered. In FIG. 6E, the entries for dataobjects C, C′ and A′ are not used, so that space is recovered in FIG.6F. All active data objects have been copied over to the shadow storagearea. The final compaction is followed by catalog adjustments andcutover to the shadow storage area as the active storage area. Theoriginal storage area 504 is available for re-use.

The above description is included to illustrate the operation of certainembodiments and is not meant to limit the scope of the invention. Thescope of the invention is to be limited only by the following claims.From the above discussion, many variations will be apparent to oneskilled in the relevant art that would yet be encompassed by the spiritand scope of the invention.

1. A computer-implemented method of managing storage of data objects toa data storage environment comprising a plurality of storage nodes, themethod comprising: storing a data object at one or more of the pluralityof storage nodes; updating a storage manager catalog that maps a dataobject identification (DOID) for the data object with an actual storagelocation of the data object; identifying a storage area to recoverstorage space; and recovering the storage space, the recovering thestorage space being implemented as compaction requests to compact thedata objects from the storage area, the compaction requests beinginterspersed with other requests to the plurality of storage nodes. 2.The method of claim 1, wherein the storage manager catalog includes acount of a number of instances of a data object that is currentlyreferenced for each of the plurality of storage nodes.
 3. The method ofclaim 1, further comprising: determining whether data objects areindicated as stale or active in the storage manager catalog, wherein arespective data object is indicated as stale when a count is equal tozero, and the count corresponds to a number of instances of therespective data object being currently referenced for each of theplurality of storage nodes.
 4. The method of claim 1, wherein the otherrequests are read requests or write requests.
 5. Thecomputer-implemented method of claim 4, further comprising: fulfillingthe read requests for active data objects from the storage area andfulfilling the write requests for new data objects using a shadowstorage area during the recovering the storage space, the shadow storagearea being different from the storage area.
 6. The computer-implementedmethod of claim 1, further comprising: throttling the compactionrequests according to quality of service (QOS) parameters, the QOSparameters indicating a percentage of a maximum number of I/O operationsper second that the compaction requests consume.
 7. Thecomputer-implemented method of claim 1, wherein the recovering thestorage space further comprises: for each data object stored in thestorage space: responsive to determining that a respective data objectis active as indicated in the storage manager catalog, copying therespective data object from the storage area to a shadow storage area;updating, in the storage manager catalog, an address of the copiedrespective data object to reflect a corresponding address in the shadowstorage area; and after copying of each data object that is determinedas active from the storage area to the shadow storage area is completed,performing a set of read requests for any of the copied each data objectfrom the shadow storage area.
 8. A system, comprising: at least oneprocessor; a memory including instructions that, when executed by the atleast one processor, cause the at least one processor to performoperations comprising: storing a data object at one or more of aplurality of storage nodes; updating a storage manager catalog that mapsa data object identification (DOID) for the data object with an actualstorage location of the data object; identifying a storage area torecover storage space; and recovering the storage space, the recoveringthe storage space being implemented as compaction requests to compactthe data objects from the storage area, the compaction requests beinginterspersed with other requests to the plurality of storage nodes 9.The system of claim 8, wherein the storage manager catalog includes acount of a number of instances of a data object that is currentlyreferenced for each of the plurality of storage nodes.
 10. The system ofclaim 8, wherein the memory includes further instructions, which furthercause the processor to perform further operations comprising:determining whether data objects are indicated as stale or active in thestorage manager catalog, wherein a respective data object is indicatedas stale when a count is equal to zero, and the count corresponds to anumber of instances of the respective data object being currentlyreferenced for each of the plurality of storage nodes.
 11. The system ofclaim 8, wherein the other requests are read requests or write requests.12. The system of claim 11, wherein the memory includes furtherinstructions, which further cause the processor to perform furtheroperations comprising: fulfilling the read requests for active dataobjects from the storage area and fulfilling the write requests for newdata objects using a shadow storage area during the recovering thestorage space, the shadow storage area being different from the storagearea.
 13. The system of claim 8, wherein the memory includes furtherinstructions, which further cause the processor to perform furtheroperations comprising: throttling the compaction requests according toquality of service (QOS) parameters, the QOS parameters indicating apercentage of a maximum number of I/O operations per second that thecompaction requests consume.
 14. The system of claim 8, wherein therecovering the storage space further comprises: for each data objectstored in the storage space: responsive to determining that a respectivedata object is active as indicated in the storage manager catalog,copying the respective data object from the storage area to a shadowstorage area; updating, in the storage manager catalog, an address ofthe copied respective data object to reflect a corresponding address inthe shadow storage area; and after copying of each data object that isdetermined as active from the storage area to the shadow storage area iscompleted, performing a set of read requests for any of the copied eachdata object from the shadow storage area.
 15. A non-transitorycomputer-readable storage medium storing instructions which, whenexecuted by one or more processors, cause the one or more processors toperform operations, comprising: storing a data object at one or more ofa plurality of storage nodes; updating a storage manager catalog thatmaps a data object identification (DOID) for the data object with anactual storage location of the data object; identifying a storage areato recover storage space; and recovering the storage space, therecovering the storage space being implemented as compaction requests tocompact the data objects from the storage area, the compaction requestsbeing interspersed with other requests to the plurality of storagenodes.
 16. The non-transitory computer-readable storage medium of claim15, wherein the storage manager catalog includes a count of a number ofinstances of a data object that is currently referenced for each of theplurality of storage nodes.
 17. The non-transitory computer-readablestorage medium of claim 15, wherein the non-transitory computer-readablestorage medium includes further instructions, which further cause theone or more processors to perform further operations comprising:determining whether data objects are indicated as stale or active in thestorage manager catalog, wherein a respective data object is indicatedas stale when a count is equal to zero, and the count corresponds to anumber of instances of the respective data object being currentlyreferenced for each of the plurality of storage nodes.
 18. Thenon-transitory computer-readable storage medium of claim 15, wherein theother requests are read requests or write requests.
 19. Thenon-transitory computer-readable storage medium of claim 18, wherein thenon-transitory computer-readable storage medium includes furtherinstructions, which further cause the one or more processors to performfurther operations comprising: fulfilling the read requests for activedata objects from the storage area and fulfilling the write requests fornew data objects using a shadow storage area during the recovering thestorage space, the shadow storage area being different from the storagearea.
 20. The non-transitory computer-readable storage medium of claim15, wherein the recovering the storage space further comprises: for eachdata object stored in the storage space: responsive to determining thata respective data object is active as indicated in the storage managercatalog, copying the respective data object from the storage area to ashadow storage area, updating, in the storage manager catalog, anaddress of the copied respective data object to reflect a correspondingaddress in the shadow storage area; and after copying of each dataobject that is determined as active from the storage area to the shadowstorage area is completed, performing a set of read requests for any ofthe copied each data object from the shadow storage area.